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Abstract 


This  study  demonstrates  that  even  a  criterion-referenced  test,  in 
which  items  are  chosen  from  a  single  content  domain,  requires  a  close 
examination  of  construct  validity.  The  Individual  Consistency  Index 
(ICI)  is  effective  in  detecting  anomalous  response  patterns  resulting 
fromsome  misconception(s) .  The  subset  obtained  by  deleting  the 
responses  which  were  spotted  by  ICI  showed  a  higher  unidimensionality. 
The  same  result  was  replicated  by  another  dataset  whose  test  was 
parallel  but  not  identical  to  the  previous  one.  Although  computerized 
error  diagnostic  programs  can  identify  misconceptions  possessed  by  a 
student  in  the  very  specific  domain  of  arithmetic,  ICI  can  be  applicable 
to  more  general  domains  and  detect  possible  candidates  to  route  to  the 
expensive  error-diagnostic  programs. 
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Spotting  Erroneous  Rules  of  Operation 
by  the  Individual  Consistency  Index 
Introduction 

Item  Response  Curve  Theory  (IRT)  has  proved  its  important  role  in 
modern  testing  practices  such  as  computerized  adaptive  testing,  in  which 
each  examinee  takes  a  different  set  of  items.  The  student's  ability 
level  is  estimated  and  located  as  a  point  on  a  continuum.  However,  a 
drawback  of  IRT  models  that  are  presently  available  in  practice  is  the 
constraint  of  unidimensionality  on  datasets  (Lord  &  Novick,  1968). 
Reckase  (1979)  warned  of  and  demonstrated  the  serious  consequences  for 
parameter  estimation  resulting  from  the  violation  of  unidimensionality. 
It  has  been  observed  that  the  scores  obtained  from  most  achievement 
tests,  unlike  ability  tests,  are  affected  by  two  or  more  latent  trait 
variables. 

Moreover,  Tatsuoka  and  Birenbaum  (1979,  1981)  showed  that  the 
dimensionaltly  of  a  dataset  obtained  from  the  middle  learning  stages, 
when  the  students  are  still  far  from  mastery,  was  multidimensional  even 
though  the  test  items  are  clearly  drawn  from  a  single  domain.  Their 
result  indicates  that  a  close  examination  of  the  construct  validity  of 
criterion-referenced  tests  is  necessary. 

Brown  &  Burton  (1978)  developed  an  error-diagnostic  model  for 
whole-number  subtraction  problems.  Their  model  "BUGGY"  showed  that 
wrong  rules  can  yield  the  correct  answers  in  some  test  items.  Birenbaum 
&  Tatsuoka  (1980)  found  that  1-0  scoring  based  simply  on  right  or 
wrong  answers  caused  serious  problems  when  erroneous  rules  of  signed- 
number  operations  were  used  by  many  examinees. 

Using  data  from  a  64-item  test  consisting  of  four  parallel  subtests 
of  16  items  each,  Birenbaum  &  Tatsuoka  first  did  a  principal  components 
analysis  on  the  original  data  —  with  the  items  scored  1  or  0  in  the 
usual  manner.  Next,  the  data  were  modified  by  giving  a  score  of  0  when 
an  item  was  correctly  answered  presumably  by  use  of  an  erroneous  rule, 
and  another  principal  component  analysis  was  done.  The  change  between 
the  two  analyses  was  dramatic.  The  dimensionality  of  the  data  became 
much  more  clearcut  with  the  modified  data.  The  item-total  correlations 
became  much  higher,  while  the  means  of  the  16  tasks  (each  represented  by 
four  parallel  items)  did  not  change  significantly.  The  above  phenomenon 
suggests  why  some  achievement  tests  cannot  by  treated  as  uni dimensional 
even  though  the  items  are  taken  from  a  single  content  domain. 

One  of  the  purposes  of  this  paper  is  to  investigate  methods  of 
extracting  a  unidimensional  subset  from  an  achievement  dataset.  Error 
analysis,  which  is  usually  performed  by  a  series  of  clinical  interviews 
and  intuitive  interpretation  of  the  student's  responses  to  the  test 
items  is  a  cumbersome  work.  At  present,  there  are  only  a  very  few 
computer  programs  available  for  providing  diagnoses  of  misconceptions 


2 


possessed  by  students,  such  as  Brown  and  Burton's  "BUGGY",  and  "SIGNBUG" 
developed  by  Tatsuoka  et  al. .  (1980).  But  they  are  expensive,  and 
moreover,  they  can  handle  only  specific  areas  of  arithmetic.  Our 
intention  is  to  find  a  more  general  technique  applicable  to  other 
content  areas  in  order  to  detect  aberrant  response  patterns  which  are 
derived  from  erroneous  rules  of  operation  (or  bugs). 

The  second  purpose  is  to  replicate  the  result  described  in  the 
Birenbaum-Tatsuoka  study  by  applying  the  general  technique  for  detecting 
aberrant  response  patterns  derived  from  wrong  rules  of  operation. 

It  turned  out  that  the  index.  Individual  Consistency  Index 
(ICI)  introduced  in  Tatsuoka  &  Tatsuoka  (1980)  is  very  effective  for 
spotting  erroneous  rules  of  operation  in  signed-number  computation 
problems:  The  responses  yielded  by  wrong  rules  are  characterized  by 
having  low  scores  and  high  ICI  values.  Moreover,  the  subset  obtained 
from  the  original  dataset  by  deleting  the  subjects  who  have  low  total 
scores  and  high  ICI  values  demonstrated  exactly  the  same  phenomenon, 
that  is,  the  dimensionality  of  the  subset  became  nearly  unldlmenslonal , 
as  the  modified  data  did.  The  structure  of  the  subset  in  terms  of 
cognitive  performance  is  Interesting.  It  consists  of  the  responses 
produced  by  using  the  right  rule  and  errors  probably  committed  by 
students  randomly  or  inconsistently.  The  result  will  be  useful  for 
understanding  the  meaning  of  dimensionality  of  achievement  data.  It 
also  shows  the  importance  of  construct  validity,  even  in  criterion 
referenced  testing  of  the  cognitive  aspect  of  performance,  and  that  the 
traditional  means  of  item  analysis  that  are  based  on  taking  the 
variances  of  binary  scores  and  content  analysis  into  consideration  are 
not  enough  for  constructing  test  items  that  are  capable  of  diagnosing 
misconceptions. 


Method  and  Procedure 
Is  1-0  scoring  justifiable? 

A  test  containing  64  signed-number  addition  and  subtraction 
problems,  consisting  of  four  parallel  subtests  of  16  items  each,  was 
administered  to  127  eighth  graders  at  a  local  junior  high  school  after 
the  Instruction  was  completed.  (This  test  will  be  referred  to  as  the 
"November  data"  hereafter.)  Each  item  in  the  test  was  carefully  related 
so  as  to  maximize  the  capability  of  diagnosing  erroneous  rules  of 
operation.  In  signed-number  computation,  98Z  of  students'  responses  are 
summarized  by  four  types:  adding  or  subtracting  two  absolute  values  and 
putting  the  sign  of  plus  or  minus  on  answers.  Nine  problem  types  in 
subtraction  and  six  in  addition  (see  Appendix  I)  are  the  necessary 
minimum  number  of  items  in  order  to  maintain  the  error  diagnostic 
capability  for  providing  a  specific  description  of  a  vast  majority  of 
popular  errors. 

Tatsuoka  et  al.  (1980)  developed  an  error-diagnostic  system  called 
"SIGNBUG"  for  signed  number  problems  on  the  PLATO  system  at  the 
University  of  Illinois.  With  this  computer  program,  the  performance  of 
the  127  students  was  thoroughly  analyzed. 
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The  same  test  was  administered  to  180  seventh  graders  who  were 
still  far  from  the  mastery  stage  and  exhibited  a  variety  of  confusion  in 
the  material.  (Data  from  this  test  will  be  referred  to  as  the  "January 
data"  hereafter).  The  responses  to  the  items  in  the  January  data  were 
also  analyzed  by  "SIGNBUG"  and  their  complete  erroneous  rules  of 
operation  with  those  from  the  November  data  are  described  in  Appendices 
1  and  II.  Actually  we  have  found  many  more  erroneous  rules,  incomplete 
ones  and  those  applicable  only  to  addition  problems,  but  the  number  of 
bugs  in  the  list  is  limited  to  the  erroneous  rules  that  appeared  in 
subtraction  problems.  The  addition  items  form  another  dimension  by  both 
a  principal  components  analysis,  and  a  multidimensional  scaling 
(Birenbaum  &  Tatsuoka,  1980)  after  the  modification  procedure  described 
earlier  was  taken.  Therefore  we  chose  the  subtraction  problems  for 
our  study. 


Insert  Table  1  about  here 


Table  1  is  the  list  of  the  binary  scores  on  15  tasks  (i.e.,  problem 
types)  yielded  by  the  45  bugs  given  in  Appendix  II.  The  rightmost 
column  shows  the  total  scores  on  15  tasks.  The  first  and  second  numbers 
in  the  parentheses  are  the  total  scores  on  addition  and  subtraction 
tasks,  respectively.  The  bottom  line  of  Table  1  contains  the  total 
number  of  l's  for  each  item  type.  For  example,  for  the  task  -16  -  (-7), 
the  correct  answer  is  yielded  by  26  out  of  42  erroneous  rules.  If  the 
data  is  collected  while  many  students  are  confused  by  a  variety  of 
errors,  then  this  task.  No.  8,  will  have  the  highest  number  of  l's. 

Thus,  although  task  No.  8  turned  out  to  be  the  "easiest"  in  the  January 
data,  this  must  be  partly  attributed  to  the  fact  that  the  correct  answer 
can  be  obtained  by  so  many  erroneous  rules.  Table  2  shows  the  rank 
orders  in  the  proportion  correct  of  12  tasks  for  the  January  and  the 
modified  November  data  (as  described  earlier).  However,  the 


Insert  Table  2  about  here 

counterpart  of  this  task,  -6  -  (-8)  has  only  twelve  l's  out  of  42. 

These  items  should  almost  be  equally  difficult  with  respect  to  the 
conceptualization  of  the  subtraction  problems  because  of  the  teaching 
method.  But  their  positions  of  the  item  difficulty  order  in  the 
datasets  (both  November  and  January  data)  are  quite  different.  The 
descending  order  of  the  total  l's  over  15  tasks  in  Table  1  is  8,  15,  7, 
16,  11,  5.  These  six  items  are  in  the  top  seven  items  having  the  largest 
number  of  the  total  l's  in  the  January  data.  As  mentioned  earlier,  80Z 
of  the  examinees  in  the  January  data  used  a  variety  of  erroneous  rules 
of  operation. 

Importance  of  Item  Ordering 

Harnisch  and  Linn  (1981)  classified  indices  that  measure  the  degree 
to  which  an  individual  response  pattern  is  atypical  into  two  different 
types.  One  type  consists  of  those  formulated  by  using  the  orders  of 


Table  1 


Observed  Complete  Rules  and  tiieir  Response  Patterns 
for  Signed  Number  Addition  and  Subtraction  Problems 


Bugs 

Addition 

Subtraction 

Total 

Scores 

3 

5 

10 

11 

14 

15 

1 

2 

4 

7 

8 

9 

12 

13 

16 

1  or  2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

15  (6,9) 

3 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

0 

1 

1 

11  (6,5) 

4 

1 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

1 

0 

11  (6,5) 

5 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

6  (6,0) 

6 

1 

1 

1 

1 

1 

1 

0 

0 

0 

1 

0 

0 

0 

0 

1 

8  (6,2) 

7  or  8 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

0 

0 

1 

10  (6,4) 

9 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

6  (6,0) 

10 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

6  (6,0) 

11 

1 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

1 

0 

11  (6,5) 

12 

1 

1 

1 

1 

1 

1 

1 

0 

1 

1 

1 

0 

1 

1 

1 

13  (6,7) 

13 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

6  (6,0) 

14 

1 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

0 

0 

10  (6,4) 

15 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

0 

1 

14  (6,8) 

16 

1 

1 

1 

1 

1 

1 

0 

0 

0 

1 

1 

0 

0 

0 

0 

8  (6,2) 

17 

1 

1 

1 

1 

1 

1 

1 

0 

0 

1 

1 

0 

0 

0 

0 

9  (6,3) 

18 

1 

1 

1 

1 

1 

1 

0 

0 

0 

1 

1 

0 

0 

0 

0 

8  (6,2) 

19 

0 

0 

1 

0 

1 

0 

0 

1 

1 

0 

0 

1 

1 

1 

0 

7  (2,5) 

20 

0 

1 

0 

1 

1 

1 

1 

0 

0 

1 

1 

0 

0 

0 

1 

8  (4,4) 

21 

1 

1 

1 

1 

1 

1 

0 

1 

0 

0 

0 

1 

0 

1 

0 

9  (6,3) 

22 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

8  (6,2) 

23 

0 

1 

0 

1 

0 

1 

0 

0 

0 

1 

1 

0 

0 

0 

0 

5  (3,2) 

24 

0 

1 

0 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

5  (3,2) 

25 

0 

1 

0 

1 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

1 

5  (3,2) 

26 

1 

0 

0 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

5  (3,2) 

27 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

3  (1,2) 

28 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

2  (0,2) 

29 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

3  (1,2) 

30 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

1 

0 

0 

0 

0 

3  (1,2) 

31 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

3  (1,2) 

32 

0 

1 

0 

0 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

3  (1,2) 

33 

0 

0 

1 

0 

1 

0 

0 

0 

0 

1 

1 

0 

0 

0 

0 

4  (2,2) 

34 

0 

0 

0 

0 

0 

1 

0 

0 

1 

0 

1 

0 

1 

0 

0 

4  (1,3) 

35 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

0 

1 

3  (1,2) 

36 

0 

0 

0 

0 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

2  (0,2) 

37 

1 

0 

0 

1 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

4  (2,2) 

38 

0 

0 

1 

1 

1 

0 

0 

1 

0 

1 

0 

1 

0 

0 

0 

6  (3,3) 

39 

0 

0 

1 

0 

1 

0 

0 

0 

0 

1 

1 

0 

0 

0 

0 

4  (2,2) 

40 

1 

1 

1 

1 

1 

1 

1 

0 

0 

1 

1 

0 

0 

0 

0 

9  (6,3) 

41 

0 

0 

0 

0 

0 

0 

1 

1 

1 

1 

1 

1 

1 

1 

1 

9  (0,9) 

42 

0 

1 

0 

1 

0 

1 

0 

1 

1 

0 

0 

1 

1 

1 

0 

8  (3,5) 

43 

1 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3  (3,0) 

44 

1 

1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

6  (6,0) 

45 

1 

1 

1 

1 

1 

1 

0 

1 

0 

0 

0 

1 

0 

0 

0 

8  (6,2) 

Total  l*s  24  28  26  30  26  32  12  10  10  20  26  10  10  9  18 


Proportion 
of  l's  to 
#  of  bugs 


_3  _7  _5  __9 
*22  22  22  22  2 


5  J_1 
“  22 


U10102026l0l0_9JL8 
42  42  42  42  42  42  42  42  42 


*22  erroneous  »  alt  (or  bugs)  because  21  out  of  43  bugs  used  the  right 
rule  for  addition  problems. 
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Table  2 

Rank  Orders  of  the  Task  Difficulties  (from  Easy  to  Hard) 


in  Two  Datasets 


Task 

Type 

November0 

Modified 

January  Data^ 

12  +  -3 

3 

7 

-14  +  -5 

9 

12 

-3  +  12 

5 

6 

-5+-7b 

13 

3 

3  +— 5 

10 

11 

-6  +  4 

14 

5 

3-6 

6 

9 

-16  -  (-7) 

7 

8 

-6  -  (-8) 

1 

1 

-3  -  +12 

12 

2 

2-11 

15 

10 

9  -  (-7)b 

11 

4 

1  -  (-10) 

4 

-7-9 

2 

-12  -  3b 

3 

g 

The  task  type 

6+4  (item  number 

6  in  Appendix  I)  has 

been  omitted. 

b These  tasks  were  note  included  in  the  January  test. 

Q 

The  November  order  is  based  on  the  difficulties  estimated 
from  the  IRT  model. 

^The  January  order  is  based  on  the  actual  proportion 
answering  each  item  correct. 
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difficulty  and  the  other  comprises  those  based  on  the  comparison  of  an 
individual  response  pattern  to  some  kind  of  a  standard  response  pattern. 
The  former  group  consists  of  Van  der  Flier's  index  (1977),  the  Norm 
Conformity  Index  (Tatsuoka  &  Tatsuoka,  1980)  and  Y  index  (Sato,  1972). 
The  latter  contains  Sato's  Caution  index  (Sato,  1975),  Linn  &  Harnisch's 
(1981)  modified  caution  index  and  Kane  &  Brennan's  (1978)  coefficient 
of  agreement.  A  weighted  sum  of  NCI  leads  to  Cliff's  (Cliff,  1978) 
Consistency  Index,  Ct2  (Tatsuoka  &  Tatsuoka,  1980)  while  the  caution 
index  has  an  algebraic  relationship  to  Loevinger's  homogeneity  index 
(Takeya,  1978). 

Although  this  classification  is  useful  for  pointing  out  a  certain 
conceptual  difference  between  the  two  types  of  indices,  the  fact  remains 
that  both  types  are  dependent  on  the  order  in  which  the  items  are 
arranged  in  calculating  them.  This  dependence  on  item  order  is  made 
explicit  in  the  case  of  the  Norm  Conformity  Index  (NCI),  whose 
definition  calls  for  arranging  the  items  in  descending  order  of 
difficulty  for  an  arbitrary  norm  or  reference  group.  The  extent  to 
which  a  given  individual's  response  pattern  then  resembles  a  Guttman 
vector  (in  which  all  zeros  precede  all  l's)  with  the  same  number  of  l's 
is  what  the  NCI  measures.  The  reference  group  may  be  the  group  of  which 
the  individual  is  a  member,  or  it  may  be  some  other  group  of  interest  to 
the  researcher. 

On  the  other  hand,  the  caution  index  was  defined  by  Sato  (1975)  in 
the  context  of  a  data  matrix  in  which  the  items  are  arranged  in 
ascending  order  of  difficulty  for  the  group  at  hand  and  the  individuals 
are  arranged  in  descending  order  of  total  score.  (Such  a  matrix  is 
called  an  S-P  table.)  The  question  of  dependence  on  item  order  does  not 
normally  arise  since  the  caution  index  Cj  is  defined  for  this  one 
particular  arrangement  of  items  only.  It  is  the  complement  of  the  ratio 
of  two  covariances  that  will  be  specified  later.  C^  measures  the  extent 
to  which  the  i-th  Individual's  response  pattern  Is  atypical  of  the  group 
of  which  he/she  is  a  member.  However,  there  is  no  reason  why  we  cannot 
speak  of  the  atypicality  of  an  individual's  response  pattern  compared  to 
the  average  response  pattern  of  some  group  other  than  the  one  to  which 
he/she  belongs.  The  order  of  items  will  then  be  different  from  before, 
and  the  value  of  Cj  will  change.  Thus  the  caution  index,  too,  is 
dependent  on  item  order. 

We  wish  to  demonstrate  the  extent  and  way  in  which  NCI  and  Cj  are 
item-order  dependent  with  reference  to  the  set  of  12  items  shown  in 
Table  2,  that  are  common  to  both  the  November  data  and  the  January  data, 
but  have  different  difficulty  orders  in  the  two  datasets.  However,  it 
may  be  useful  first  to  give  brief  descriptions  of  the  calculations  for 
the  two  indices  by  using  a  smaller  numerical  example. 

Example.  Let  us  refer  back  to  the  43  x  9  matrix  of  binary  scores 
obtained  by  using  the  correct  rule  and  42  erroneous  rules  for  solving 
nine  subtraction  problems,  displayed  in  the  right-hand  panel  of  Table  1. 
We  now  pretend  that  this  is  a  data  matrix  for  a  group  of  43  examinees 
who  took  a  nine-item  test,  and  denote  it  as 
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Y  -  (yij),  1-1,2.. .  ,43;  j-l,2,...,9 

Each  element  of  the  vector  of  column  totals  [y.i,  y.2****  y . 9 1  is  one 
greater  than  the  corresponding  column  total  shown  in  Table  1  (because 
the  totals  there  excluded  the  l's  in  the  first  row  since  that  row 
represented  the  correct  rule  of  operation,  whereas  Table  1  was  concerned 
with  l's  generated  by  erroneous  rules).  However,  since  we  need  to 
arrange  the  items  in  monotonic  order  of  difficulty  (descending  order  for 
NCI,  ascending  for  C^),  let  us  renumber  the  items  from  1  to  9  in  the 
order  they  are  to  occur  in  the  formulas. 

For  calculating  NCI,  then,  the  item  order  is  (2,  9,  13,  12,  4,  1, 
16,  7 ,  8) .  We  take  the  second  row  (for  Bug  #3)  as  the  response  pattern 
of  the  examinee  whose  NCI  we  want  to  calculate.  When  the  items  are 
rearranged  as  just  indicated,  the  response  vector  becomes 

I3  ■  [1  1  1  0  0  0  1  I  0]  , 

whose  elements  we  denote  as  y3i,  y32»***»y39  in  the  order  they  occur 
here. 

We  also  need  the  total  score  y3#,  which  is  5  in  this  case. 

In  general. 


yi.  "  J^ij 

where  n  (■  9  here)  is  the  number  of  items.  The  NCI  for  Individual  i  may 
now  be  defined  as 


(NCI)*  -  2Ula/Ui  -  1  , 

where 

n-1  n 

°u  *j!i  kV1'),13>,tk 

and 

ui  -  yi.<n-yi.) 

[Verbally,  Uia  means  the  sum  of  all  l's  to  the  right  of  each  0  in  the 
vector  yi,  added  over  all  0's;  this  represents  the  number  of  (0,1) 
pairs  that  occur  in  U^,  on  the  other  hand,  represents  the  total 

number  of  (0,1)  and  (1,0)  pairs  in  jj^.J 
For  our  example. 


U3a  -  2  +  2  +  2  -  6 

(because  there  are  three  0's,  each  with  two  l's  to  its  right;  the  last  0 
does  not  have  any  number  to  its  right  and  hence  contributes  nothing  to 
the  sum).  While 
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U3  -  <5 ) ( 4 )  -  20  . 

Hence , 

(NCI)3  -  (2)(6)/20  -  1  -  -.4 

(The  negative  sign  indicates  that  the  response  pattern  g3  is  closer  to 
the  reverse  Guttman  vector  [111110000]  than  to  the  Guttman 
vector  [000011111],  which  agrees  with  our  impression  on  looking 
at  y3.) 

We  now  illustrate  the  calculation  of  q  for  the  same  response 
pattern.  Since  the  formula  calls  for  the  items  to  be  arranged  in 
ascending  order  of  difficulty,  we  reverse  the  response  vector  to  get 

j3  ■  [0  1  1  0  0  0  1  1  1] 

We  also  need  the  vector  of  column  totals 

in  -  fy.i»  y.9] 

-  [26,  20,  18,  12,  11,  10,  9,  9,  21 
Finally,  the  reverse  Guttman  vector  with  five  l's  is 

i3s  -  [1  1  1  l  1  0  0  0  0] 

With  the  elements  of  these  three  vectors  appropriately  symbolized, 
and  '3'  replaced  by  i  for  generality,  the  formula  for  the  caution  index 
is 

q  -  1  -  ^v-lyiJ >  y-J> 

cov  (yij,  y.j) 

1  Making  the  substitutions,  we  get 

9  9 

cov(y3j,  y.j)  -  -  y3.  j2,y.j/9]/8 

-  [(0)(26)  +  (1)(20)  +  (1)(18)  +  ...  +  (1)(2) 

-(5) (26  +  20  +  18  +  ...  +  2)/9]  /8 

I  -  -.875 

i 

! 

9  9 

cov(y3j,  y.j)  -  [^ySjy.j  -  y$.  ^ 


and 
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-  [(0(26)  +  (0(20)  +  (008)  +  (002)  +  (OOO  +  0 
-(5) (26  +20+18  +...+  2 ) / 9 ] 8 
=  2.75 

Hence,  C3  =  1  -  (-.875)/2.75  =  1.318 

Item-Difficulty  Orders  In  Early  and  Late  Learning  Stages. 

We  now  examine  the  response  patterns  of  three  students  taken  from 
the  January  dataset,  exhibit  their  NCI  and  values  with  items  arranged 
in  difficulty  order  for  that  dataset  as  well  as  in  difficulty  order  for 
the  November  dataset,  and  note  the  extent  and  nature  of  the  differences 
between  the  corresponding  values.  Although  the  two  datasets  are  based 
on  different  samples,  the  results  nevertheless  give  a  general  idea  of 
the  effect  of  using  the  difficulty  orders  in  early  and  late  learning 
stages.  The  relevant  information  is  summarized  in  Tables  3  and  4. 

Insert  Table  3  about  here 


Student  #37  Is  a  very  good  student,  who  did  most  of  the  addition 
problems  correctly,  as  seen  in  Table  4.  About  the  only  trouble  she  had 
was  confusing  parentheses  with  absolute-value  bars,  which  is  a 
relatively  minor  and  easily  remedied  misconception.  Yet,  precisely 
because  her  misconception  Is  a  sophisticated  one  for  students  at  this 
stage  of  learning,  her  response  pattern  is  rare  and  atypical  of  the 
group.  She  thus  gets  a  low  NCI  (-.10)  and  a  high  C^  (.93),  which  are 
misleading  because  they  imply  that  she  needs  to  be  cautioned  and  given 
remedial  work. 


Insert  Table  4  about  here 


On  the  other  hand,  Students  #12  and  #30  possess  misconceptions  that 
are  rather  common  in  this  dataset.  Hence,  their  NCI  values  are 
relatively  high  (.62  and  .69,  respectively)  while  their  caution  indices 
are  moderate  and  low(.43  and  .16),  so  that  these  students  are  not 
flagged  for  further  attention.  Yet,  these  students  (like  many  others  in 
the  January  dataset)  had  considerable  trouble  with  addition  problems,  as 
evident  from  Table  4.  Therefore,  their  not  being  cautioned  was 
Inappropriate,  and  was  due  only  to  their  errors'  being  fairly  common  for 
their  group. 

To  confirm  the  above  interpretations,  let  us  now  look  at  the  NCI 
and  Ci  values  for  these  three  students  when  the  items  are  arranged  in 
difficulty  order  for  the  November  dataset  as  modified  in  the  manner 
described  earlier.  That  is,  we  now  inquire  how  anomalous  each  of  these 
students'  response  patterns  would  look  if  they  had  been  members  of  the 
November  group,  who  were  close  to  mastery  stage.  The  answer  is,  the  NCI 
for  Student  #30  becomes  the  largest  of  three  (.67),  while  Students  #37 
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Table  3 


Response  Patterns  of  Pour  Subtests  for  Students  if 37,  if  12  and  #30 


January  Order 

S#37 

S#12 

S#30 

Nov.  Modified 

S#37 

S#12 

S#30 

-16  -  (-7) 

8 

0000 

1110 

nil 

12  +  -3 

3 

nil 

1010 

nil 

2-11 

16 

0100 

1110 

0000 

-14  +  -5 

10 

nil 

0010 

0000 

12  +  -3 

3 

mi 

1010 

nil 

-3  +  12 

5 

ion 

mi 

mi 

8-6 

D 

0100 

nil 

nil 

3  +  -5 

11 

nil 

nil 

0111 

-6+4 

B 

ini 

1101 

nil 

-6  +  4 

15 

nil 

1101 

nil 

-3  +  12 

B 

ion 

nil 

nil 

8-6 

7 

0100 

nil 

mi 

3  +  -5 

i 

ii 

nil 

nil 

0111 

-16  -  (-7) 

8 

0000 

1110 

ini 

-14  +  -5 

10 

ini 

0010 

0000 

-6  -  (-8) 

1 

0000 

1010 

1000 

-6  -  (-8) 

1 

0000 

1010 

1000 

-3  -  +12 

13 

0111 

0100 

0000 

1 

1 

vC 

B 

nil 

0000 

0000 

2-11 

16 

0100 

1110 

0000 

-3  -  +12 

B 

0111 

0100 

0000 

1  -  (-10) 

1 

4 

0000 

0000 

0000 

1  -  (-10) 

B 

0000 

0000 

0000 

1 

1 

vC 

2 

nil 

0000 

0000 

Total  Score 

28 

27 

24 

28 

27 

24 

Caution  Index 

.93 

.43 

.16 

.30 

.60 

.44 

NCI 

-.10 

.62 

.69 

.49 

.44 

.67 

ICI 

.96 

• 

00 

00 

.99 

.96 

00 

CO 

» 

.99 

Table  4 


Error 

Patterns 

for  Subject  #37,  #12 

and  #30 

Subject 

Addition 

Subtraction 

Subt  1 

Subt  2 

Subt  3 

Subt  4 

Subt  1 

Subt  2 

Subt  3 

Subt  4 

S# 

37 

(11,21) 

(11,0) 

(11,21) 

(11,21) 

treat  parentheses 

as  absolute  value 

S# 

12 

(13,21) 

(13,0) 

0 

(0,21) 

(13,0) 

(0,24) 

(13,0) 

0 

s# 

30 

(13,0) 

(13,21) 

(13,21) 

(13,21) 

(13,0) 

*(13,21) 

*(13,21) 

*(13,21) 

*  -  (32) 
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and  12  have  similar  values  (.49  and  .44).  Their  caution  indices  are 
.30,  .60  and  .44  for  Students  #37,  12  and  30,  respectively.  As  expected 
#37  has  the  smallest  caution  index,  reflecting  the  fact,  already 
mentioned,  that  her  errors  were  more  typical  for  students  close  to  the 
mastery  stage  than  the  beginners  in  the  January  dataset. 

The  foregoing  illustrative  examples  should  have  underscored  the 
undesirability  (for  error-diagnostic  purposes)  of  the  dependence  of  such 
indices  as  NCI  and  on  the  particular  difficulty  order  of  items  that 
is  used  in  computing  them.  Clearly,  what  is  needed  is  an  index  that 
does  not  depend  on  the  item  difficulty  order  for  a  group  but  on  that  for 
each  individual  in  his/her  own  right. 


The  Individual  Consistency  Index  (ICI) 

The  Individual  Consistency  Index  (ICI)  depends  on  the  task 
difficulties  as  determined  by  an  individual  student's  state  of 
knowledge.  Its  definition  calls  for  the  existence  of  two  or  more 
parallel  subtests,  and  the  sets  of  parallel  items  are  arranged  in 
ascending  order  of  task  difficulties  for  the  student  in  question  — 
i.e.,  in  the  order  of  the  student's  subscores  on  the  separate  tasks  such 
as  the  12  shown  in  Table  3,  where  each  task  is  represented  by  four 
parallel  items.  The  actual  calculation  of  ICI  is  the  same  as  that  for 
NCI  —  or,  more  precisely,  for  a  group  of  m  subjects,  where  m  Is  the 
number  of  parallel  subtests. 

Its  value  is  large  to  the  extent  that  the  individual  remains  in  the 
same  state,  and  hence  responds  to  similar  items  in  the  same  way  —  i.e., 
using  the  same  rule.  If  ICI  is  low,  on  the  other  hand,  then  the 
individual  is  probably  not  sure  how  the  problems  can  be  solved  so  he/ she 
tries  various  strategies  to  respond  to  the  questions.  Or,  he/she  is 
careless  and  makes  a  considerable  number  of  random  errors.  Thus,  ICI 
can  serve  better  than  NCI  and  the  Caution  Index  as  an  index  for  flagging 
individuals  who  probably  require  further  attention  for  remediating  their 
errors  and/or  making  finer  diagnoses. 


Table  5  is  a  summary  of  the  error  patterns  committed  by  the  127 
students  in  the  November  data,  divided  into  four  subgroups  by  the 
following  criteria:  (1)  students  whose  scores  are  higher  than  52  (the 
highest  total  score  in  Table  1  is  13.  Therefore  4  x  13  is  taken  as  a 
criterion)  and  ICI  values  are  greater  than  or  equal  to  .90;  (2)  those 
whose  scores  are  lower  than  or  equal  to  52  and  ICI  values  are  greater 
than  or  equal  to  .90;  (3)  those  who  have  scores  higher  than  or  equal  to 
52  and  have  ICI 


scores 


52 


JL 


tk 
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values  lower  than  .90;  (4)  chose  whose  scores  are  lower  chan  or  equal  co 
52  and  ICI  values  are  lower  chan  .90. 


InserC  Table  5  about  here 


The  error  Cypes  and  cheir  frequencies  are  shown  In  four  secdons  of 
Che  Cable.  As  sCaCed  in  Che  Cable,  a  number  sign  (#)  in  front  of  an 
error  pattern  represents  the  scores  adjusted  for  both  addition  and 
subtraction  problems  (Bfrenbaum  &  Tatsuoka,  1980).  The  question  marks 
in  front  of  the  error  patterns  stand  for  the  scores  adjusted  for 
subtraction  problems,  even  though  rules  were  equivocal.  The  detailed 
description  of  adjusting  criterion  is  given  in  Birenbaum  &  Tatsuoka.  The 
point  is  that  the  modified  dataset  Improved  the  approach  to 
unidimensionality  drastically.  Moreover,  parameter  estimation  for  the 
two  parameter  logistic  model  by  maximum  likelihood  (using  the  computer 
program  GETAB  written  by  Robert  Balllie)  was  processed  successfully  with 
the  modified  dataset  while  the  original  dataset  failed  to  yield  finite 
estimates. 

It  is  interesting  to  note  that  Category  2  contains  most  erroneous 
patterns  with  high  consistency.  On  the  other  hand.  Categories  1  and  3 
consist  of  patterns  in  which  the  right  rule  appears  at  least  once  and 
the  other  elements  are  zeros.  The  zeros  mean  that  either  the  response 
patterns  are  Inconsistent  or  that  rules  are  so  complicated  that  our 
error-diagnostic  system  could  not  determine  them  specifically.  But  from 
the  error  patterns,  and  a  close  examination  of  the  generated  error 
vectors  in  our  error-diagnostic  system,  the  zeros  in  Categories  1  and  2 
are  mostly  due  to  random  errors.  Therefore  we  can  safely  conclude  that 
the  structure  of  the  unidimensional  subset  consists  of  the  responses 
that  result  from  using  the  right  rule,  plus  random  errors. 

Replication  of  the  Result  Obtained  from  the  November  Data 

The  November  data  were  obtained  from  8th  graders  in  1979.  A  year 
later,  a  new  group  of  161  8th  graders  took  a  64-item  signed-number 
addition  and  subtraction  test  that  was  parallel  to  the  November  test. 

The  test  was  administered  right  after  the  teachers  (who  also  taught  the 
students  in  the  November  data)  completed  their  Instruction.  The 
teachers  used  exactly  the  same  teaching  method,  materials  and  quizzes  as 
those  of  the  last  year.  The  only  difference  in  the  new  data,  the  March 
data,  is  that  the  numbers  in  the  16  tasks  are  slightly  changed  as  shown 
in  Appendix  V.  Table  6  is  a  summary  of  the  classification  of  error 
patterns  by  ICI  and  total  scores. 


Insert  Table  6  about  here 

The  result  again  Indicates  that  ICI  is  a  useful  index  for 
extracting  a  unidimensional  dataset  by  classifying  the  erroneous 
patterns  into  Category  2.  Also  the  data  structure  of  the  extracted 
unidimensional  subset  in  Table  6  confirms  that  the  responses  consist  of 
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Table  5 


Error  Patterns  Causing  a  Mess  in  Dimensionality 
of  Nine  Subtraction  Tasks  (36  items) 


Category  1 

Category 

2 

Error  Pattern  Frequency 

Error  Pattern 

Frequency 

(1)  (1)  (1)  (1) 

15 

(5)  (5)  (5)  (5) 

4 

0  (1) (1) (1) 

5 

0  (5)  (5)  (5) 

1 

(1)  0  (1)(1) 

3 

it 

0  0  (5) (5) 

1 

(1) (1)  0  (1) 

2 

it 

(16) (16) (40) (16) 

1 

(1) (1) (1)  0 

2 

(5)  (40)  (5)  (5) 

1 

0  0  <1> (1) 

2 

it 

(16)  (16)  (5)  (16) 

1 

0  (1)(1)  0 

1 

# 

(16) (16)  (16)  0 

1 

(1)  0  (1)  0 

1 

it 

(16) (40) (16) (16) 

1 

0  (1)  0  (1) 

1 

it 

(40) (40) (40) (40) 

2 

(1)  0  0  0 

2 

(40)  0  (5) (5) 

i 

0  (1)  0  0 

1 

it 

0  (40) (16) (5) 

i 

0  0  (1)  0 

3 

it 

(5) (40) (40) (40) 

i 

(1)  o  0  (1) 

1 

it 

(40) (40) (40) (16) 

2 

(40)  (1)  (1)  (1) 

1 

it 

0  (40) (16) (16) 

1 

0  0  0  0 

1 

it 

(40) (40) (23) (23) 
(19) (19) (19)  (19) 

0  (19)  (19)  (19) 

1 

1 

1 

Total  No.  of  Stu. 

41 

it 

it 

0  0  (40) (40) 

1 

#Scores  adjusted 

0  (40)  0  0 

1 

it 

(6) (6) (6) (6) 

1 

*Performance  on  addition 

it 

(15) (15) (15) (15) 

2 

problems  was  not  consistent 

it 

(4)  (4)  (4)  (4) 

1 

it 

(7) (7) (7) (7) 

1 

?Scores  adjusted  for  sub 

- 

it 

(43) (43) (43) (43) 

2 

traction  problems,  even 

it 

0  0  0  (5) 

1 

LllUUgll  luxe  WCIC  C^UXVUCclX 

Total  No.  of  Stu. 

32 

Category  3 

Category  4 

Error  Pattern  Frequency 

Error  Pattern  Frequency 

(1)  (1) (1) (1) 

53 

0  0  0  0 

7 

0  (1) (1) (1) 

5 

? 

0  0  0  0 

1 

(1)  0  (1) (1) 

4 

? 

0  0  0  (1) 

1 

(1) (1)  0  (1) 

2 

? 

0  0  (1)  0 

1 

(1)(1)(1)  0 

1 

0  (1)  0  0 

1 

0  0  (1) (1) 

3 

? 

0  0  0  (5) 

1 

0  (1)  0  (1) 

3 

0  0  (23) (23) 

1 

(1)  CD  o  o 

2 

it 

0  0  (40) (40) 

1 

0  (1) (1)  0 

1 

it 

0  0  0  (19) 

1 

(1)  0  (1)  0 

1 

0  0  (n)(i) 

1 

0  (1)  0  0 

1 

0  (1)  0  (1) 

1* 

0  0  0  (1) 

4 

it 

0  (1) (1) (1) 

1* 

0  0  (1)  0 
(40)  0  0  0 

1 

1 

1 

0  (1) (1) (1) 

1 

Total  No.  of  Stu. 

15 

U  0  0  0 

Total  No.  of  Stu. 

5 

(1)  means  that  the  right  rule  is  used  for  answering  to  subtraction  problems. 

aSince  1C1  is  calculated  over  16  tasks,  if  performance  on  addition  problems 
is  not  consistent,  then  the  error  pattern  falls  into  Category  3  even  though 
the  error  pattern  for  subtraction  is  Identical. 
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Table  6 

Replication  of  the  result  obtained  froa  the  Noveaber  Data: 
N  “  161,  Classification  of  Bugs 


0  0  0  0  2 

(1) (11) (1) (11)  1 


Total  Ho.  of  Stu.  68 


Category  2 

Error  Pattern  Frequency 


(5)  (5)  (5)  0  2 

0  (5) (5) (5)  1 

0  0  (5)  (5)  1 

0  0  (40)  0  1 

(40)  0  (40)  0  1 

0  (40) (40)  0  2 

0  0  0  (40)  2 

(40) (16) (5) (5)  1 

(40) (16) (40) (5)  1 

0  (5) (16) (16)  1 

0  (16) (16)  0  1 

0  0  0  0  6 

(19) (19) (19) (19)  1 

0  0  (19)  (19)  1 

0  (19) (19)  0  2 

0  (3) (3) (3)  1 

(11)  (11) (11) (11)  1 

0  (43) (43) (43)  1 

(4) (4) (4)  (4)  1 

(4)  0  0  (4)  1 


Total  Ho.  of  Stu.  29 


Categoi 


Error 

Pattern 

Frequency 

Error 

Pattern 

(1) (1) (1) 

(1) 

32 

(1) 

(1) 

(1)U) 

0 

(1) 

(1) 

(1) 

7 

(1)(1) 

0 

(1) 

(1) 

0 

(1) 

(1) 

5 

0 

(1) 

(D(l) 

(1) 

(1) 

0 

(1) 

6 

(1)  (1) 

0 

0 

(1)(1) 

(1) 

0 

4 

0 

(1) 

(l) 

0 

(1) 

(1) 

0 

0 

1 

0 

0 

(1) 

(1) 

(1) 

0 

(1) 

0 

1 

(1) 

0 

(1) 

0 

(1) 

0 

0 

(1) 

1 

(1) 

0 

0 

(1) 

0 

(1) 

(1) 

0 

3 

(1) 

0 

0 

0 

0 

0 

(1) 

(1) 

2 

0 

(1) 

0 

0 

0 

(1) 

0 

(1) 

1 

0 

0 

(1) 

0 

(1) 

0 

0 

0 

1 

0 

0 

0 

(1) 

0 

0 

0 

(1) 

1 

0 

0 

0 

0 

Frequenc 


Total  No.  of  Stu. 


Categoi 
Error  Pattern 


0  (1)  0  (1) 

0  0  0  (1) 
000  (40) 
0  0  (40)  0 
0  (5)  0  (5) 

0  0  0  0 


Frequenc 


1 

2 

1 

2 

1 

10 


Total  No.  of  Stu. 
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answers  gotten  by  using  the  right  rule  and  of  random  errors,  which  is 
the  identical  result  obtained  from  the  November  data. 


Insert  Table  7  about  here 


Table  7  shows  the  eigenvalues  of  the  March  data  before  and  after 
ICI  operation  was  applied.  The  unidimensionality  of  the  subset  improved 
but  not  as  much  as  that  of  the  November  data.  Since  the  proportion  of 
the  number  of  subjects  in  Category  2  is  1/4  in  the  November  data  while 
it  is  1/8  in  the  March  data,  the  difference  in  the  increments  of  the 
variance  accounted  for  by  the  first  eigenvalues  in  the  two  datasets  may 
be  explained.  The  March  data,  both  the  original  and  those  modified  by 
ICI  yielded  convergence  when  GETAR  was  used  to  estimate  the  parameters 
of  the  IRT  model. 


Conclusion 

One  of  the  important  >reas  in  the  theory  and  practice  of 
educational  measurement  if.  .hut  of  criterion-referenced  testing.  The 
measurement  theories  chat  nave  been  utilized  in  measuring  and  evaluating 
the  outcomes  of  treatments  (or  Instruction)  typically  depend  on  binary 
scores  obtained  from  test  items.  It  was  customarily  assumed  that  the 
underlying  structure  of  fifse  dataset  from  a  criterion-referenced  test 
consists  of  one  major  common  factor  because  the  items  are  usually 
selected  from  a  single  content  domain.  However,  several  studies  have 
shown  that  the  assumption  of  unidimensionality  should  be  closely 
examined  even  for  criterion-referenced  tests.  This  study  demonstrated 
that  the  dimensionality  of  a  dataset  will  be  affected  by  the  examinee's 
underlying  cognitive  processes  as  well  as  by  the  nature  of  the  content 
domain.  The  fact  is  that,  after  the  response  patterns  yielded  by 
erroneous  rules  of  operation  are  deleted  from  the  original  dataset,  the 
remaining  subset  of  data  becomes  more  nearly  unldlmenslonal ,  and  this 
subset  of  course  consists  of  responses  yielded  by  the  right  rule  and 
non-systematic  errors.  This  observation  points  to  an  answer  to  the 
question  of  when  and  why  the  dataset  of  an  achievement  test  satisfies 
the  condition  of  unidimensionality. 

Tables  5  and  6  show  that  ICI  is  an  effective  index  for  detecting 
erroneous  rules  of  oeration.  However,  it  requires  repeated  measures. 
Since  most  tests  don't  have  several  parallel  items  to  measure  the 
performance  of  a  single  task,  applicability  of  ICI  to  these  tests  will 
be  limited.  The  drawback  of  this  limitation  mus  be  removed,  and  a 
solution  to  this  problem  is  being  developed. 
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Table  7 

Eigenvalues  and  their  variances  of  the  replicated  datasets 


The  Original  Dataset 
H  -  161 


Subset  in  which  low  score-high 
ICI  subjects  are  deleted.  N  ■  137 


Factor 

Eigenvalues 

Variance 

Eigenvalue 

Variance 

1 

9.2928 

58.0802 

9.4820 

59.2627 

2 

1.7834 

11.1462 

1.3967 

8.7295 

3 

1.0883 

6.8018 

0.9202 

5.7511 

4 

0.8437 

5.2729 

0.8070 

5.0439 

5 

0.6985 

4.3659 

0.5943 

3.7141 

6 

0.4301 

2.6879 

0.5155 

3.2217 

7 

0.3728 

2.3303 

0.4466 

2.7911 

8 

0.3331 

2.0819 

0.3959 

2.4742 

9 

0.2428 

1.5178 

0.2905 

1.8155 

10 

0.1857 

1.1606 

0.2807 

1.7543 

11 

0.1747 

1.0918 

0.2344 

1.4649 

12 

0.1458 

0.9115 

0.1806 

1.1286 

13 

0.1312 

0.8197 

0.1585 

0.9906 

14 

0.1172 

0.7323 

0.1171 

0.7318 

15 

0.0868 

0.5423 

0.0999 

0.6243 

16 

0.0731 

0.4569 

0.0803 

0.5018 
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Appendix  I 

The  Signed-Number  Test  used  in  the 
January  Experiment 


I 

11 

Test  Items 

III 

IV 

1. 

-6-(-8)=2 

17. 

— 1— <— 1 0 ) =9 

33. 

-3-(-5)=2 

49. 

— 2— ( — 11) =9 

2. 

-7-9= — 16 

18. 

-2-1 1 =— I 3 

34. 

-4-6— 10 

50. 

-5-14—19 

3. 

12+-3=9 

19. 

7+— 5=2 

35. 

15+— 6=9 

51. 

4+-2=2 

A. 

l-(-10 )  =  1 1 

20. 

3— (— 1 2 ) =1 5 

36. 

5-(-7)=12 

52. 

6-(-8)=14 

5. 

-3+1 2=9 

21. 

-1+10=9 

37. 

-4+13=9 

53. 

-2+11=9 

7. 

8-6=2 

23. 

7-5=2 

39. 

4-2=2 

55. 

9-7=2 

8. 

-16- (-7)— 9 

24. 

— 12— C~ 10)=— 2 

40. 

-1 l-(-2)=-9 

56. 

-7-(-5)=-2 

10. 

-1A+— 5=-19 

26. 

-10+-1=-11 

42. 

-7+-5=-12 

58. 

-10+-8—18 

11. 

3+-5=-2 

27. 

2h — 1 1=-9 

43. 

6+— 8=-2 

59. 

H — 10—9 

12. 

13-(-4)=l 7 

28. 

0-(-9)=9 

44. 

6— (—4 )=1 0 

60. 

0-(-2)=2 

13. 

-3-+12— 15 

29. 

-2-+1 1 =— 1 3 

45. 

-7-+9=-16 

61. 

-4— +6—10 

15. 

-6+A=-2 

31. 

— 5+3=—  2 

47. 

-4+2=-2 

63. 

-8+6—2 

16. 

2-1 l=-9 

32. 

5-14=-9 

48. 

7—1 6=— 9 

64. 

4-13—9 

i 

i 

r 
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Appendix  II 


A  List 

of  "Bugs"  Found  in  the  Sign-numbered  Addition  and 

Subtraction  Problems  (Verbal  Rules) 

Code 

Incomplete  Rule  (in  terms  of  sign  operation) 

1. 

21 

Taking  the  sign  of  the  larger  absolute  number  to  answers 
(i.e.,  take  the  correct  sign  in  addition  problems.) 

2. 

22 

Taking  the  sign  of  the  smaller  absolute  number  to  answers 

3. 

23 

Taking  always  a  +  sign  to  answers 

4. 

24 

Taking  always  a  -  sign  to  answers 

5. 

25 

Taking  the  sign  of  the  first  number 

6. 

26 

Taking  the  sign  of  the  second  number 

7. 

27 

Taking  the  sign  of  the  product  of  the  two  numbers 

8. 

28 

Taking  the  sign  of  the  larger  number  (as  an  integer)  to 
answers 

9. 

29 

Taking  the  sign  of  the  smaller  number  (as  an  integer)  to 
answers . 

10. 

210 

Treat  the  operation  sign  as  the  sign  of  the  second 
number,  i£  the  sign  of  the  second  number  is  explicxt. 

Then  take  the  sign  of  the  larger  absolute  values  to 
answers . 

11. 

211 

Besides  the  bug  described  in  210,  if  the  sign  of  the 
larger  absolute  value  is  the  first  number  with  an 
implicit  sign,  then  take  the  explicit  sign  or  the  sign 
obtained  by  the  210  bug  to  answers 

12. 

212 

The  same  as  bug  210  except  take  the  sign  of  the  larger  integer. 

13. 

213 

Basically  follow  the  right  rule,  but  when  the  sign  of  the 
larger  absolute  value  is  implicit  and  also  is  the  second 

number,  then  take  the  sign  of  the  other  number  to  answers. 


(cont. ) 
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Appendix  II  (cont.) 


Code  Incomplete  Rule  (in  terms  of  sign  operation) 


14.  214 


15.  215 


16.  216 

17.  217 

18.  218 

19,  219 


Basically  follow  the  bug  of  code  28,  but  when  the  larger 
integer  is  the  second  number  and  its  sign  is  implicit, 
then  take  the  sign  of  the  other  number. 

For  the  addition  problems,  if  the  sign  of  the  second  number 
is  implicit,  then  apply  the  regular  arithmetic  operation. 

For  example,  -L+S  has  a  wrong  sign.  For  the  subtraction 
problems,  the  implicit  sign  of  the  second  number  remains 
unchanged  when  they  converted  to  addition  problems,  and 
regular  arithmetic  operation  is  applied  but  -L+S  type  has 
a  wrong  sign. 

Change  the  explicit  sign  of  the  second  number,  then  take 
the  sign  of  the  larger  absolute  value. 

Change  the  explicit  sign  of  the  second  number,  then  take 
the  sign  of  the  larger  integer 

Change  the  sign  of  the  second  number  and  take  the  sign  of 
the  larger  absolute  value.  Treat  all  problems  as 
subtraction  problems. 

Change  the  sign  of  the  second  number  and  take  the  sign  of 
the  larger  integer  value.  Treat  all  problems  as  subtraction 
problems. 
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Appendix  II  (cont.) 


Code 

Incomplete  Rules  (in  terms  of  absolute  value  operation) 

1. 

11 

If  the  signs  of  two  numbers  are  same,  then  add  the  absolute 
values  of  the  two  numbers.  If  the  signs  of  two  numbers 
are  different,  then  subtract  the  smaller  absolute  value 
from  the  larger  absolute  value 

2. 

12 

In  spite  of  different  skill  types,  the  absolute  values  of 
the  two  numbers  are  always  added. 

3. 

13 

No  matter  what  the  skill  types  are,  the  smaller  absolute 
value  is  subtracted  from  the  larger  absolute  value. 

4. 

14 

Opposite  of  the  right  operation,  i.e.,  if  the  signs 
of  two  number  are  same,  then  subtract,  |l|  -  |s|. 

If  the  signs  of  two  numbers  are  different  then  add  the 
absolute  values  of  the  two  numbers,  |l|  +  |s|. 

5. 

15 

If  the  sign  of  the  first  number  is  positive,  +,  then 
|L |  +  |S|. 

If  the  sign  of  the  first  number  is  negative,  -,  then 
|L|  -  |S|. 

6. 

16 

If  the  sign  of  the  second  number  is  positive,  +,  then 
|L|  +  |s|  and  if  the  sign  of  the  second  number  is 
negative,  then  |L|  -  |s|. 

7. 

17 

If  the  sign  of  the  first  number  is  positive,  then 
| L.  |  -  | S  j  and  if  it  is  negative,  then  |l|  +  |S|. 

8. 

18 

If  the  absolute  value  of  the  first  number  is  larger,  then 
|L|  +  | S |  and  if  it  is  smaller,  then  |l|  -  |s|  . 

9. 

19 

If  the  absolute  value  of  the  first  number  is  larger, 
then  | L |  +  | S | . 

10. 

110 

If  the  first  number  is  larger  as  an  integer,  then 
| L |  +  | S | .  If  it  is  smaller  then  |l|  -  |s|. 

11. 

111 

If  the  first  number  is  larger  as  an  integer,  then  |l|  -  |s| 
and  if  its  not  then  add  the  two  absolute  values. 

12. 

112 

Changing  the  sign  of  the  second  number,  then  applying 
the  right  rule  11. 
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Appendix  II  (cont.) 

Subtraction  Problems 

(Problems  in  the  conversion  of  subtraction  to  addition) 


Code 


I. 

31 

Convert  the  operation  sign  -  to  +  and  change  the  sign 
of  the  second  number. 

2. 

32 

Convert  the  operation  sign  -  to  +  and  don  't  change  the 
sign  of  the  second  number 

3. 

33 

Convert  the  operation  sign  -  to  +  and  change  the  sign  of 
the  first  number 

4. 

34 

Convert  the  operation  sign  -  to  +  and  change  both  the 
signs  of  the  two  numbers 

5. 

35 

Convert  the  operation  sign  -  to  +  and  change  the  sign  of 
the  second  number.  At  the  same  time,  if  the  sign  of  the 
first  number  is  negative,  -,  then  it  will  be  changed  to  + 

6. 

36 

Convert  the  operation  sign  -  to  +  but  don't  change  the 
sign  of  the  second  number.  At  the  same  time,  if  the  sign 
of  the  first  number  and  second  numbers  are  negative,  then 
change  the  signs  to  +.  Thus,  task  -L  -  (-S)  is  converted 
to  +L  +  (+S) . 
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Appendix  II  (cont.) 

Sub tract ion  Problems 
Having  Hidden  Signs  and  Parentheses 
(Interpretation  of  the  problems) 


hs  When  the  sign  of  the  second  number  is  implicit,  the  sign 

of  the  second  number  is  ignored.  Thus,  the  sign  of  the 
second  number  remains  the  same  although  the  operation 
sign  is  changed  to  This  error  appears  in  all 

4  types  of  problems,  L-S,  S-L,  -L-S  and  -S  -  L. 

ah  In  the  above  mentioned  case,  L-S  and  S-L  are  carried 

out  by  regular  arithmetic  but  -L-S  and  -S-L  will  be 
converted  to  -L  +  S  and  -S  +  L  respectively. 

a  L-S  and  S-L  are  treated  as  regular  arithmetic  problems. 

The  implicit  signs  of  the  skill  types  -L  -  S,  -S  -  L 
are  not  ignored.  So  they  are  changed  to  -L  +  -S 
-S  +  -L  respectively 

Pbf  The  parentheses  of  the  problems  are  interpreted  as  absolute 

value  symbols  before  any  conversion  of  subtraction  to 
addition. 

Paf  The  parentheses  are  treated  as  absolute  value  symbols  after 

any  conversion  occurs. 

ai  Besides  L-S  and  S-L  types,  students  apply  regular 

arithmetic  operations  to  (-L)  -  (-S)  and  (-S)  -  (-L) 
types  and  get  wrong  signs  for  S-L  and/or  (-S)  -  (-L). 

ho  The  operation  sign  in  the  task  -L  -  S  is  recognized  as  the 

sign  of  the  second  number.  So  when  a  student  converts  sub¬ 
traction  problems  to  addition  problems,  insert  a  +  sign  before 
tne  second  minus  sign,  right  after  the  larger  number  L.  So  this 
task  will  yield  the  task  type,  -L  +  -S. 
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Appendix  III 

Observed  Complete  Erroneous  Rules  of  Operations 
Their  Descriptions  and  Codes  Given  In  Appendix  H 


1.  ( 1 1 ,21 )(31 )( 1 1 ,21 ) 

The  right  rule  for  addition  problems,  (11,21)  —  add  the  absolute 
values  of  two  numbers  if  the  signs  of  the  numbers  are  same,  or  subtract 
the  smaller  absolute  value  from  the  larger  one  if  the  signs  are  alike 
and  take  the  sign  of  the  larger  number  to  the  answer.  Then  convert 
subtraction  problems  into  addition  ones  by  changing  the  operation  sign 
-to  4-  and  the  sign  of  the  second  number  to  a  opposite  sign.  Carry  out 
the  right  addition  operation  on  the  newly  converted  add! ton  problems. 

2.  ( 1 1,21 ) (31 )Paf(l 1,21) 

The  right  rules  for  addition  problems,  and  right  conversion  (31)  is 
carried  out  but  the  numbers  in  the  parenthesis  are  changed  to  positive 
numbers,  then  the  right  rule  for  addition  is  used  for  the  new  addition 
problems . 

3.  (11 ,21)Pbf (31)(11 ,21) 

The  numbers  in  the  parenthesis  are  changed  to  positive  before 
subtraction  problems  are  converted  to  addition  problems. 

A.  (11,25)(31)(11,25) 

The  student  takes  the  right  absolute  value  in  answers  for  addition 
problems.  For  subtraction  problems,  the  student  converts  subtraction 
to  addition  correctly  but  applies  the  same  wrong  rule. 

5.  (11, 21)(32)(11, 21) 

The  right  rule,  for  taking  the  proper  absolute  value  and  the  sign 
of  the  larger  absolute  value,  is  used  for  addition  problems.  For 
subtraction  problems  a  student  changes  the  operation  sign  of  -  to  4- 
wlthout  changing  the  sign  of  the  second  number  then  applies  the  right 
rule  for  addition  problems.  The  code  is  expressed  by  (11,21)  for 
addition,  conversion  error  (32)  and  the  right  rule  (11,21)  again.  Thus, 
this  bug  is  (11,21)(32)(11,21). 

6.  ( 1 1,21 ) (32 )a( 11,21) 

The  addition  problems  are  right,  but  subtraction  problems  are 
converted  in  a  wrong  way  —  by  changing  operation  signs,  -,  to  plus,  4-, 
except  for  the  problems  types  L  -  S  and  S  -  L.  The  latter  two  tasks  are 
answered  by  a  regular  arithmetic  method.  The  other  converted  tasks  are 
answered  by  using  the  right  rule. 

7.  (1 1,21 )(32)ha( 11,21) 

Apply  the  right  addition  rule  and  converts  most  subtraction 
problems  by  applying  (32)  —  changes  operation  signs  to  plus  but  doesn't 
change  the  signs  of  the  second  number  —  but  -L  -  S  type.  Operation  4- 
is  Inserted  right  after  the  larger  number  L.  So  the  problem  becomes 
-L  4-  -S  type. 


8.  ( 1 1,21 )(32)hoa( 11,21) 

This  bug  is  a  combination  of  bugs  (ll,21)(32)a(ll  ,21)  and 
(ll,21)(32)ho(ll,21).  That  is,  -L  -  S  type  was  changed  to  -L  +  -S,  and, 
moreover,  L  -  S  and  S  -  L  types  were  answered  by  a  regular  arithmetic 
method. 

9.  (11 ,21 )( 36) (11,21) 

After  all  subtraction  problems  are  converted  to  addition  problems 
according  to  the  rule  (36)-— convert  the  operation  sign  -  to  +, without 
changing  the  sign  of  the  second  number.  At  the  same  time,  if  there  are 
three  minuses  such  as  -L-(-S),  then  all  minuses  will  be  plus.  Thus 
+L+C+S)  is  the  converted  problem  type. 

10.  (11,21)(33)(11,21) 

Convert  subtraction  problems  into  addition  problems  but  change  the 
sign  of  the  first  number  instead  of  the  second  number.  The  right 
addition  rule  was  applied  before  and  after  the  conversion. 

11.  ( 1 1,21 ) (31 )hs(l 1,21) 

When  subtraction  problems  are  converted  to  addition,  hidden  signs 
of  the  second  numbers  are  Ignored,  so  they  are  not  changed  to  negative. 
Thus  -L  -  S ,  -S  -  L,  S-L,  L  -  S  resulted  in  wrong  answers. 

12.  ( 1 1,21 ) (31 )hsa( 11,21) 

L  -  S  and  S-L  types  are  answered  by  a  regular  arithmetic  method 
without  being  converted,  but  -  L  -  S  and  -S-L  are  converted  to  -  L  +  S 
and  -S  +  L,  resulting  in  wrong  answers. 

13.  ( 1 1,21 )(32)Pbf (11,21) 

Before  subtraction  problems  are  converted,  the  parenthesis  in  the 
problems  are  considered  as  an  absolute  value  notation  and  the  numbers  in 
the  parenthesis  are  changed  to  positive  numbers  before  the  conversion 
(32)  is  taken  . 

14.  (ll,21)(32)Paf(ll,21) 

The  numbers  in  the  parenthesis  are  changed  to  be  positive  numbers 
after  subtraction  problems  are  converted  to  addition  problems  according 
to  the  wrong  rule  (32). 

15.  (ll,21)(32)hoPaf (11 ,21) 

-L  -  S  and  -S-L  types  resulted  in  right  conversion  even  though 
the  wrong  conversion  rule  (32)  was  applied.  Then  the  problems  with 
parenthesis  are  changed  to  be  positive. 

16.  (11 ,21)03,21)  {or  (11,21)(32)(13,21)} 

A  student  used  the  right  rule  for  addition  problems  but  he/she 
subtracted  two  numbers,  |L|  -  |S|,  and  took  the  sign  of  the  larger 
absolute  value. 


17.  (ll,21)Paf(13,21) 

After  carrying  out  the  rule  13,  subtracting  two  numbers,  the 
numbers  In  the  parenthesis  are  changed  to  positive  numbers  then  the  rule 
(21)  is  used  for  taking  the  sign  to  answers. 

18.  (ll,21)ho(13,21) 

For  -L  -  S  type,  having  a  hidden  sign  for  the  second  number,  a 
student  treated  operation  sign  -  as  the  sign  of  the  second  number. 

19.  (12,21)(31)(12,21) 

Subtraction  problems  are  converted  to  addition  problems  by  the 
right  conversion  rule  but  an  erroneous  rule  for  addition  problem, 

(12.21)  are  applied  consistently  both  before  and  after  the  conversion. 

20.  (13,21)(31)(13,21) 

The  operation  of  converting  subtractin  to  addition  is  carried  out 
correctly  but  a  wrong  rule  (13,21) —  always  subtracting  two  numbers,  |L| 
-  | S |  and  taking  the  sign  of  the  larger  absolute  value —  was  applied 
throughout  the  problems. 

21.  (11,21)02,24) 

The  conversion  of  subtraction  was  not  carried  out.  For  addition, 
the  right  rule  was  used,  but  for  subtraction  problems,  two  numbers  are 
added  and  a  minus  sign  was  taken  to  the  answers. 

22.  (11,21)03,24) 

The  conversion  of  subtraction  was  not  carried  out.  The  rule 

(11.21)  for  addition,  the  rule  (13,24)  for  subtraction  problems. 

23.  (13,21)03,21) 

The  conversion  of  subtraction  was  omitted  and  two  numbers  are 
always  subtracted  and  the  sign  of  the  larger  absolute  value  was  always 
taken  to  answers. 

24.  (13,21)03,24) 

The  conversion  was  omitted  and  erroneous  rules  (13,21)  for  addition 
problems,  (13,24)  for  subtraction  problems  are  used. 

25.  (13,21)(32)a(ll,21) 

For  addition  problems,  the  wrong  rule  (13,21)  is  used.  The 
conversion  is  again  wrong  (32).  L  -  S  and  S  -  L  are  answered  by  a 
regular  arithmetic  method.  The  rest  of  the  newly  converted  addition 
problems  are  answered  by  the  right  rule 

26.  (13,24)03,24) 

The  same  rule,  |L|  -  |S|  and  the  sign,  -,  to  answers. 

27.  (12,24)03,24) 

The  conversion  operation  is  ignored.  Add  two  numbers  for  addition, 
subtract  two  numbers  |L|  -  |S|  for  subtraction  problems.  The  sign,  -, 
is  always  taken  to  answers. 
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28.  (12,23X13,24) 

No  conversion  Is  made.  The  two  numbers  are  added  for  addition 
problems  and  subtracted  for  subtraction  problems.  The  sign,  +,  is 
always  taken  to  answers  for  addition  while  the  sign,  -,  is  always  taken 
for  subtraction  problems. 

29.  (13,23X13,24) 

No  conversion  is  made.  The  two  numbers  are  always  subtracted,  |L| 

-  | S |  but  +  for  addition  problems  and,  -,  for  problems  are  taken  to 
answers. 

30.  (13,25X13,25) 

No  conversion  is  made.  The  two  numbers  are  always  subtracted, 
t L |  -  | S |  and  the  sign  of  the  first  number  is  taken  to  answers. 

31.  (13,25)03,24) 

No  conversion  is  made.  The  two  numbers  are  always  subtracted  |L| 

-  | S | .  The  sign  of  the  first  number  is  taken  to  answers  for  addition 
problems  and  a  minus  sign  is  always  taken  for  subtraction  problems. 

32.  (13,23)03,23) 

No  conversion  is  made.  The  two  numbers  are  always  subtracted,  |L| 

-  | S| .  The  sign,  +,  is  always  taken  to  answers. 

33.  (12,25)03,25) 

No  conversion  is  made.  The  two  numbers  are  added  for  addition  and 
subtracted  for  subtraction  problems.  The  sign  of  the  first  numbber  is 
taken  to  answers  for  both  the  addition  and  subtraction  problems. 

34.  (15,25)05,25) 

The  operation  of  converting  subtraction  to  addition  problems  is 
ignored.  Instead,  this  student  added  two  numbers,  |L|  +  |S|,  if  the 
sign  of  the  first  number  is  positive.  He  subtracted,  |L|  -  |S|  if  the 
sign  of  the  first  number  is  negative.  He  used  this  rule  for  both  the 
addition  and  subtraction  problems. 

35.  (15,25)03,24) 

For  addition  problems,  the  same  rule  as  the  previous  example  is 
applied  but  for  subtraction  problems,  he  subtracts  two  numbers,  |L|  -  |S| 
and  takes  a  minus  sign. 

36.  (16,23)03,23) 

For  addition  problems,  if  the  sign  of  the  second  number  is  positive 
then  two  absolute  values  are  added,  |L|  +  |S|  and  if  it  is  negative, 
then  the  two  absolute  values  are  subtracted,  |L|  -  |S|.  For 
subtraction  problems,  the  two  numbers  are  always  subtracted  and  a 
positive  sign,  +,  is  always  chosen  for  the  sign  to  answers. 


37.  (16,26)hoU6,26) 

For  addition  problems,  the  absolute  values  of  answers  are  the  same 
as  the  previous  example  of  (16,23).  But  the  sign  of  the  answers  is  the 
sign  of  the  second  number.  For  subtraction  problems,  the  same  rule  as 
addition  is  used,  but  -L  -  S  or  -S  -  L  types  where  the  sign  of  the 
second  number  is  implicit,  then  operation  sign  is  considered  as  the  sign 
of  the  second  number. 

38.  (17,21X17.21) 

For  addition  and  subtraction  problems,  if  the  sign  of  the  first 
number  Is  +,  then  |L|  -  ] S I  and  if  it  is  negative,  then  |L|  +  |S|.  But 
the  sign  of  the  larger  absolute  value  is  taken  to  answers. 

39.  (12, 21)(13,21) 

For  addition  problems,  absolute  values  of  the  two  numbers  are 
always  added  and  the  sign  of  the  larger  number  is  taken  to  answers.  For 
subtraction  problems,  two  absolute  values  are  always  subtracted  and  the 
sign  of  the  larger  number  is  taken  to  answers. 

40.  (12,21)Ai  (13,21) 

The  right  rule  for  addition  problems.  For  subtraction  problems, 

L  -  S,  S  -  L,  -L  -  (-S)  and  -S  -  (-L)  types  are  answered  by  using  a 
regular  arithmetic  method  but  the  signs  of  answers  are  not  always  correct. 
All  other  problems  are  answered  by  (13,21)-rule.  No  conversion 
of  subtraction  to  addition  problems  was  made. 

41.  (32) (11,21) (31) (11,21) 

Subtraction  problems  are  correctly  converted  by  the  rule  (31),  and 
answered  correctly  by  using  the  right  rule  of  addition  problems.  But 
addition  problems  are  also  converted  by  changing  the  sign  of  the  second 
number  and  answered  by  the  right  rule. 

42.  (13,21)(31>(12,21) 

In  addition  problems,  the  student  finds  the  difference  between  the 
two  numbers  and  assigns  the  sign  of  the  larger  absolute  value  to  the 
result.  In  subtraction,  converts  subtraction  to  addition  by  Rule  (31) 
which  is  right,  then  adds  the  two  absolute  values  and  takes  the  sign  of 
the  larger  absolute  value  from  the  converted  addition  problems  to 
answers. 

43.  (11,27)03,24) 

If  two  numbers  have  two  similar  signs,  then  the  two  absolute  values 
are  added  and  sign  of  multiplication  is  taken  to  the  result.  If  two 
numbers  have  different  signs,  then  |L|  -  |S|  as  the  absolute  value  and  a 
minus  sign  to  the  answer. 

44.  (11, 21) (34) (11,21) 

In  subtraction,  the  student  changes  the  signs  of  both  numbers  as 
well  as  the  operation  sign  and  carries  out  the  right  rule  for  addition 
problems . 

45.  (11,21)06)03,21) 

In  subtraction,  the  two  absolute  values  are  subtracted  and  the  sign 
of  the  larger  number  is  taken  to  the  result  after  conversion  rule  (36) 
is  applied. 
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Appendix  V 

The  Signed-Uumber  Test 
The  Marcn  data 
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